Existing techniques for working with data tend to provide a satisfactory user experience in most cases. For example, a user interacting with a database may use a graphical user interface (GUI) to issue various queries to a back-end database. In many scenarios, the queries can be executed in short periods of time and the user can continue working with the database interactively. This is also the case for many development scenarios, e.g., a software developer may write various queries, test them against the database, and receive results in an interactive fashion. This facilitates query development because the software developer can revise a given query as needed if the query does not produce expected results.
However, for large data sets, existing techniques do not always provide a satisfactory user experience. For example, “big data” tools such as Apache™ Hadoop® enable analysis of massive amounts of data, but not necessarily in an interactive fashion. In a big data context, a given query may take a very long time to complete, perhaps even overnight. From the perspective of the software developer responsible for writing the query, it can be difficult to debug such queries efficiently because the software developer waits a long time until the query completes before evaluating the results for correctness.